support for auto enabling gpu tests by smuzaffar · Pull Request #2825 · cms-sw/cms-bot

smuzaffar · 2026-08-12T15:59:32Z

No description provided.

smuzaffar · 2026-08-12T15:59:39Z

please test

cmsbuild · 2026-08-12T16:00:10Z

A new Pull Request was created by @smuzaffar for branch master.

@akritkbehera, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-08-12T16:00:11Z

cms-bot internal usage

smuzaffar · 2026-08-12T16:11:59Z

please test with cms-sw/cmssw#51689

cmsbuild · 2026-08-12T16:17:05Z

Pull request #2825 was updated.

cmsbuild · 2026-08-12T16:22:41Z

Pull request #2825 was updated.

cmsbuild · 2026-08-12T16:37:29Z

Pull request #2825 was updated.

smuzaffar · 2026-08-12T19:14:44Z

please test

cmsbuild · 2026-08-12T19:15:16Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55288/summary.html
COMMIT: 7b3b892
CMSSW: CMSSW_20_0_X_2026-08-12-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55288/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testLLVMAnalyzers had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 2641501
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2641483
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 168 log files, 141 edm output root files, 44 DQM output files
TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 7 DQM output files
TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

AMD_W7900 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 7 DQM output files
TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 7 DQM output files
TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_L4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 7 DQM output files
TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 7 DQM output files
TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_T4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 7 DQM output files
TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

cmsbuild · 2026-08-13T12:26:29Z

Pull request #2825 was updated.

cmsbuild · 2026-08-13T13:01:42Z

Pull request #2825 was updated.

smuzaffar · 2026-08-13T13:43:44Z

please test with cms-sw/cmssw#51691

cmsbuild · 2026-08-13T15:06:41Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55312/summary.html
COMMIT: 677f2e1
CMSSW: CMSSW_20_1_X_2026-08-13-0800/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55312/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3435544
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3435526
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
Checked 191 log files, 159 edm output root files, 44 DQM output files
TriggerResults: no differences found

smuzaffar · 2026-08-13T15:51:06Z

please test with cms-sw/cmssw#51691

cmsbuild · 2026-08-13T15:51:32Z

Pull request #2825 was updated.

cmsbuild · 2026-08-13T18:38:13Z

-1

Failed Tests: RelVals-AMD_W7900
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55321/summary.html
COMMIT: 2eca285
CMSSW: CMSSW_20_1_X_2026-08-13-0800/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55321/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_W7900

37634.40437634.404_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling/step2_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3435544
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3435526
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
Checked 191 log files, 159 edm output root files, 44 DQM output files
TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 121 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 9835
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 163750
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 90 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 8992
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164593
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 82 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 8596
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164989
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 97 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173585
DQMHistoTests: Total failures: 8098
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165487
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff 235.0 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_L4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff -39.0 exceeds +/- 30.0 MiB

smuzaffar · 2026-08-14T09:29:13Z

please test

cmsbuild · 2026-08-14T11:41:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45c27/55339/summary.html
COMMIT: 2eca285
CMSSW: CMSSW_20_1_X_2026-08-13-2300/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2825/55339/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3591807
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3591780
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: no differences found

cmsbuild added orp-pending pending-signatures externals-pending tests-started labels Aug 12, 2026

cmsbuild added requires-external tests-pending and removed tests-started requires-external labels Aug 12, 2026

cmsbuild added tests-started requires-external and removed tests-pending labels Aug 12, 2026

support for auto enabling gpu tests

bdc5121

smuzaffar force-pushed the auto-gpu-tests branch from 416a694 to bdc5121 Compare August 12, 2026 16:22

cmsbuild added tests-pending and removed tests-started requires-external labels Aug 12, 2026

add gpus in ENABLE_BOT_TESTS too

7b3b892

smuzaffar mentioned this pull request Aug 12, 2026

[Do not merge] Testing bot for auto-enabled gpu tests cms-sw/cmssw#51691

Draft

cmsbuild added tests-rejected and removed tests-pending tests-rejected labels Aug 12, 2026

cmsbuild added the tests-started label Aug 13, 2026

smuzaffar mentioned this pull request Aug 13, 2026

[do not merge] testing cms-bot cms-sw/cmsdist#10784

Draft

bug fix

5c1e70a

cmsbuild added tests-pending and removed tests-started labels Aug 13, 2026

use correct xml file path

677f2e1

cmsbuild added tests-started requires-external and removed tests-pending labels Aug 13, 2026

cmsbuild added tests-approved and removed tests-started labels Aug 13, 2026

unset scram env

2eca285

cmsbuild added tests-started and removed tests-approved labels Aug 13, 2026

cmsbuild added tests-rejected and removed tests-started labels Aug 13, 2026

cmsbuild added tests-started and removed requires-external tests-rejected labels Aug 14, 2026

cmsbuild added tests-approved and removed tests-started labels Aug 14, 2026

Uh oh!

Conversation

smuzaffar commented Aug 12, 2026

Uh oh!

smuzaffar commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Uh oh!

smuzaffar commented Aug 12, 2026

Uh oh!

cmsbuild commented Aug 12, 2026

Failed Unit Tests

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L4 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Uh oh!

cmsbuild commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Comparison Summary

Uh oh!

smuzaffar commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Uh oh!

cmsbuild commented Aug 13, 2026

Failed RelVals-AMD_W7900

Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L4 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Max Memory Comparisons exceeding threshold NVIDIA_H100

Max Memory Comparisons exceeding threshold NVIDIA_L4

Uh oh!

smuzaffar commented Aug 14, 2026

Uh oh!

cmsbuild commented Aug 14, 2026

Comparison Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cmsbuild commented Aug 12, 2026 •

edited

Loading